Papers with English dataset
Polysemy through the lens of psycholinguistic variables: a dataset and an evaluation of static and contextualized language models (2024.starsem-1)
Copied to clipboard
| Challenge: | Polysemes are words that can have different senses depending on context . traditionally, NLP models assume that each sense should be given a separate representation in a lexicon, thus limiting the amount of evidence that can be gained from their use. |
| Approach: | They propose a framework to model polysemes as a continuous variation in psycholinguistic properties of a word in context without postulating jumps between senses. |
| Outcome: | The proposed framework accommodates different sense interpretations, without postulating clear-cut jumps between senses. |
ParsFEVER: a Dataset for Farsi Fact Extraction and Verification (2021.starsem-1)
Copied to clipboard
Majid Zarharan, Mahsa Ghaderan, Amin Pourdabiri, Zahra Sayedi, Behrouz Minaei-Bidgoli, Sauleh Eetemadi, Mohammad Taher Pilehvar
| Challenge: | Existing methods for fact-checking and verification require large amounts of annotated data, but this is limited to low-resource languages. |
| Approach: | They present a first publicly available Farsi dataset for fact extraction and verification . they use the construction procedure of the standard English dataset for the task . |
| Outcome: | The proposed dataset improves on the standard English dataset and is available on github. |
Revisiting Low-Resource Neural Machine Translation: A Case Study (P19-1)
Copied to clipboard
| Challenge: | Recent research has shown that neural machine translation models are highly data-inefficient and underperform phrase-based statistical machine translation (PBSMT) in low-resource settings. |
| Approach: | They propose to use auxiliary data to train low-resource neural machine translation systems without auxiliary monolingual or multilingual data. |
| Outcome: | The proposed methods outperform PBSMT and other statistical machine translation models in Korean–English with minimal data. |
GUM-SAGE: A Novel Dataset and Approach for Graded Entity Salience Prediction (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for graded entity salience are subjective but lack consistency. |
| Approach: | They propose a method for graded entity salience that combines subjective judgments and summarization-based methods that define saliency as mention-worthiness in a summary. |
| Outcome: | The proposed approach outperforms existing methods and shows stronger correlation with human summaries and alignments. |
Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to translate sentences to other languages using heuristics are challenging. |
| Approach: | They propose a model that learns hierarchical weights for different sets of labels and applies them to other languages to translate them. |
| Outcome: | The proposed model can translate English datasets to other languages and obtain different sets of labels again using heuristics. |
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2024.acl-long)
Copied to clipboard
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, Madian Khabsa
| Challenge: | Existing benchmarks for text comprehension only cover 30 languages, but lack of labeled data is a major obstacle to building functional systems in most languages. |
| Approach: | They present a multiple-choice machine reading comprehension dataset spanning 122 languages . they use it to evaluate the capabilities of multilingual masked language models and large language models . |
| Outcome: | The proposed dataset enables the evaluation of text models in high-, medium- and low-resource languages. |
Compositional Evaluation on Japanese Textual Entailment and Similarity (2022.tacl-1)
Copied to clipboard
| Challenge: | Despite growing interest in linguistic universals, most NLI/STS studies focus on English. |
| Approach: | They propose a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. |
| Outcome: | The proposed datasets show that pre-trained language models are insensitive to word order and case particles. |
Our kind of people? Detecting populist references in political debates (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing literature on populism has only limited agreement on its exact properties . |
| Approach: | They propose a cross-lingual dataset to identify populist rhetoric in text . they propose 'hierarchical' annotation procedure to annotate populist references . |
| Outcome: | The proposed dataset can be used to investigate how political actors talk about The Elite and The People and to study how populist rhetoric is used as a strategic device. |
Czech Dataset for Cross-lingual Subjectivity Classification (2022.lrec-1)
Copied to clipboard
| Challenge: | Using the existing English dataset, we can use the subjectivity classification to test the ability of pre-trained multilingual models to transfer knowledge between languages. |
| Approach: | They propose to use a Czech subjectivity dataset of 10k manually annotated subjective and objective sentences as a cross-lingual benchmark. |
| Outcome: | The proposed dataset is the first subjectivity dataset for the Czech language and also includes 200k automatically labeled sentences. |
Unlocking Large Audio-Language Models for Interactive Language Learning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Computer-Assisted Pronunciation Training (CAPT) systems provide unintuitive feedback that lacks actionable guidance. |
| Approach: | They propose to use audio-language models to provide more user-friendly feedback for pronunciation training. |
| Outcome: | The proposed model outperforms baselines on mispronunciation detection and suggestion generation. |
Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing models memorize procedures from context and rely on shallow heuristics to solve MWPs. |
| Approach: | They propose a contrastive learning approach where the neural network perceives the divergence of patterns. |
| Outcome: | The proposed method greatly improves performance in monolingual and multilingual settings. |
Discourse Self-Attention for Discourse Element Identification in Argumentative Student Essays (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite its importance, discourse element identification is challenging due to the ambiguity of sentences . the number of elaboration sentences could be 10 times more than the number edna sentences. |
| Approach: | They propose to use sentence positional encodings to explicitly represent sentence positions and inter-sentence attentions to capture sentence interactions and enhance sentence representation. |
| Outcome: | The proposed model improves on a Chinese and English dataset. |
Building a Japanese Document-Level Relation Extraction Dataset Assisted by Cross-Lingual Transfer (2024.lrec-main)
Copied to clipboard
| Challenge: | Document-level Relation Extraction (DocRE) is the task of extracting all semantic relationships from a document. |
| Approach: | They propose to transfer an English document to Japanese to promote DocRE in other languages. |
| Outcome: | The proposed model reduces the human edit steps by 50% compared with the previous approach. |
Unsupervised Paraphrasability Prediction for Compound Nominalizations (2022.naacl-main)
Copied to clipboard
| Challenge: | Nominalizations can be difficult to interpret because of ambiguous semantic relations between deverbal noun and its arguments. |
| Approach: | They propose to over-generate clausal paraphrases to predict whether a prenominal modifier can be re-written as a noun or adverb in a claual paraphrasability. |
| Outcome: | The proposed method improves paraphrasability prediction and paraphrase generation in English . it shows that the prenominal modifier can be re-written as a noun or adverb in a clausal paraphrase . |
Cross-Lingual UMLS Named Entity Linking using UMLS Dictionary Fine-Tuning (2022.findings-acl)
Copied to clipboard
| Challenge: | a new method for named entity linking is being developed in the field of public health . it uses an offline unsupervised construction of a translated dictionary and a pre-trained transformer language model to filter candidates according to context. |
| Approach: | They propose a method for mapping mentions in a source language to UMLS concepts . they extend an offline unsupervised translation of a translated UMLS dictionary . |
| Outcome: | The proposed approach achieves state-of-the-art on the Hebrew Camoni corpus and English datasets. |
Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog (2022.naacl-main)
Copied to clipboard
| Challenge: | Task-oriented dialog (TOD) is arguably one of the most popular natural language processing (NLP) application areas. |
| Approach: | They propose a multilingual multi-domain TOD dataset that spans four languages . they use a framework for multilingual conversational specialization of pretrained language models . |
| Outcome: | The proposed datasets show that they perform better than existing datasets in English . the proposed framework allows for sample-efficient few-shot transfer for TOD tasks . |
HateCOT: An Explanation-Enhanced Dataset for Generalizable Offensive Speech Detection via Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Social media has amplified the propagation of hateful sentiments, highlighting the contested nature of "offensive content" research shows that "of offensive content" is still a contested construct due to varying definitions and labeling. |
| Approach: | They propose a dataset that features human-curated explanations for offensive content in English . they show that HateCOT pretraining improves performance of open-source LLMs . |
| Outcome: | The proposed model improves on three benchmark datasets for offensive content detection . the model improve the quality of its explanations, as confirmed by the human evaluation . |
Multilingual prediction of Alzheimer’s disease through domain adaptation and concept-based language modelling (N19-1)
Copied to clipboard
Kathleen C. Fraser, Nicklas Linz, Bai Li, Kristina Lundholm Fors, Frank Rudzicz, Alexandra König, Jan Alexandersson, Philippe Robert, Dimitrios Kokkinakis
| Challenge: | Existing work on speech and language models has been limited by the size of available datasets. |
| Approach: | They propose to augment a small French dataset with a much larger English dataset to augment the language model to model the order in which information units are produced by dementia patients and controls. |
| Outcome: | The proposed model improves classification performance in English and French separately. |
IMPLI: Investigating NLI Models’ Performance on Figurative Language (2022.acl-long)
Copied to clipboard
| Challenge: | Understanding figurative language is a difficult area in NLP but is essential for proper understanding. |
| Approach: | They propose to use a dataset to generate 24k semiautomatic pairs and manually create 1.8k gold pairs to evaluate NLI models. |
| Outcome: | The proposed models can detect entailment relationship between figurative phrases and their literal counterparts, but perform poorly on similar structured examples. |
Constructing Taxonomies from Pretrained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | A variety of NLP tasks use taxonomic information, including question answering and information retrieval. |
| Approach: | They propose a method for constructing taxonomic trees using pretrained language models by incorporating web-retrieved glosses into the model. |
| Outcome: | The proposed model achieves 66.7 ancestor F1, a 20.0% relative increase over the previous best published model on English WordNet. |
Barch: an English Dataset of Bar Chart Summaries (2022.lrec-1)
Copied to clipboard
| Challenge: | a new dataset of human-written summaries of bar charts is available in english . a chart summary is a textual description of a data point, which is often analytical . |
| Approach: | They propose a dataset of human-written summaries describing bar charts in english . a total of 47 charts are presented in the dataset, which includes 47 charts . |
| Outcome: | a new dataset of human-written summaries describing bar charts is presented in english . the dataset shows that human speakers often include such statements into chart summary . |
On the Effectiveness of Adversarial Robustness for Abuse Mitigation with Counterspeech (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work on automated counterspeech systems focused on synthetic data but rarely looked into how the public deals with abuse. |
| Approach: | They propose to curate a new dataset of abuse and replies from footballers for study of public figure abuse and use it to examine how models can handle adversarial attacks. |
| Outcome: | The proposed model is robust against adversarial attacks across domains and can handle abuse in the real world. |
Multitasking Framework for Unsupervised Simple Definition Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing definition generation tasks require a dictionary with complex definitions and a corpus containing arbitrary simple texts to generate them. |
| Approach: | They propose a multitasking framework SimpDefiner that only requires a standard dictionary with complex definitions and a corpus containing arbitrary simple texts. |
| Outcome: | The proposed framework outperforms the baseline model by a 1.77 SARI score on the English dataset, and raises the proportion of the low level (HSK level 1-3) words in Chinese definitions by 3.87%. |
Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment (P19-1)
Copied to clipboard
| Challenge: | Existing models for speaker commitment fail to generalize to diverse linguistic constructions, highlighting directions for improvement. |
| Approach: | They evaluate two state-of-the-art speaker commitment models on the CommitmentBank . they analyze linguistic correlates of model error on a naturalistic dataset . |
| Outcome: | The proposed models perform well on some classes but fail to generalize to diverse linguistic constructions. |
PhotoshopQuiA: A Corpus of Non-Factoid Questions and Answers for Why-Question Answering (L18-1)
Copied to clipboard
| Challenge: | Community Question Answering web sites are used for non-factoid question answering . however, there is a scarcity of available datasets for this task . cnn.com's john m. sutter is releasing a dataset for why-QA . |
| Approach: | They propose a dataset of 2,854 why-question and answer(s) pairs related to Adobe Photoshop usage from five CQA web sites. |
| Outcome: | The new dataset is the first English dataset for Why-QA that focuses on a product . it can be used to build Why-Q systems, evaluate approaches and develop new models . |
A Large Multilingual and Multi-domain Dataset for Recommender Systems (L18-1)
Copied to clipboard
| Challenge: | Existing algorithms for recommending items are limited and focused on specific domains. |
| Approach: | They propose a multi-domain interests dataset to train and test Recommender Systems . the english dataset includes an average of 90 preferences per user on music, books, movies, celebrities, sport, politics . |
| Outcome: | The proposed method exploits popular services such as Spotify, Goodreads and others to extract preferences from Twitter messages in Italian and English. |
Harnessing the linguistic signal to predict scalar inferences (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Bayesian game-theoretic models of pragmatic reasoning can predict the strength of scalar inferences by using linguistic features. |
| Approach: | They propose to use a sentence encoder to predict the strength of scalar inferences by using a corpus of linguistic data. |
| Outcome: | The proposed model infers previously established associations between linguistic features and inference strength, suggesting that it learns to use linguistic feature to predict pragmatic inferences. |
Diverse and Informative Dialogue Generation with Context-Specific Commonsense Knowledge Awareness (2020.acl-main)
Copied to clipboard
| Challenge: | Generative dialogue systems tend to produce generic and boring responses, causing boring conversations . a novel commonsense knowledge-aware dialogue generation model is proposed to solve this problem . |
| Approach: | They propose to retrieve and introduce knowledge facts from knowledge graphs to reduce boring conversations . they use a Felicitous Fact mechanism to help the model focus on context-relevant knowledge facts . |
| Outcome: | The proposed model outperforms the state-of-the-art approach in most experiments. |
Highly Parallel Autoregressive Entity Linking with Discriminative Correction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to EL have been shown to be effective for both Entity Disambiguation and Entity Linking, but they suffer from high computational cost due to a complex (deep) decoder and the need for training on a large amount of data. |
| Approach: | They propose a method that parallelizes autoregressive linking across all potential mentions and relies on a shallow and efficient decoder. |
| Outcome: | The proposed method outperforms state-of-the-art approaches on the English dataset AIDA-CoNLL and is >70 times faster and more accurate than the previous generative method. |
Few-Shot Semantic Dependency Parsing via Graph Contrastive Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing graph neural networks (GNNs) have shown promising performance on semantic dependency parsing (SDP) training a high-performing model requires a large amount of labeled data and it is prone to over-fitting in the absence of sufficient labele . |
| Approach: | They propose a syntax-guided graph contrastive learning framework to train GNNs with unlabeled data and fine-tune pre-trained GNN models with few-shot labeled SDP data. |
| Outcome: | The proposed framework achieves promising results when few-shot training samples are available. |
Ecologically Valid Explanations for Label Variation in NLI (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Human label variation exists in many natural language processing tasks, including NLI . |
| Approach: | They build an English dataset of 1,415 ecologically valid explanations for 122 MNLI items . they find that people can systematically vary on their interpretation . |
| Outcome: | The proposed dataset contains 1,415 ecologically valid explanations for 122 items . the results show that people can vary on interpretation and highlight differences . |
MedDialog: Large-scale Medical Dialogue Datasets (2020.emnlp-main)
Copied to clipboard
Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, Hongchao Fang, Penghui Zhu, Shu Chen, Pengtao Xie
| Challenge: | telemedicine is a medical practice that provides patient care remotely using video conferencing tools. |
| Approach: | They build large-scale medical dialogue datasets to facilitate research . they pretrain several models on the Chinese MedDialog dataset and compare their performance . |
| Outcome: | The proposed datasets show that models trained on MedDialog can generate doctor-like medical dialogues. |
Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications (2020.lrec-1)
Copied to clipboard
| Challenge: | A speech technology exhibits demographic bias when performance is worse for one demographic group relative to another. |
| Approach: | They create an English dataset of expert-validated audio, transcript> pairs with demographic tags for age, gender, accent and open software which may be used to detect demographic bias in Automatic Speech Recognition systems. |
| Outcome: | The Artie Bias Corpus is a curated subset of the Mozilla Common Voice corpus, which is released under a Creative Commons CC0 license . |
Investigating the Saliency of Sentiment Expressions in Aspect-Based Sentiment Analysis (2023.findings-acl)
Copied to clipboard
| Challenge: | In a set of masking experiments, we examine the extent to which the tokens identified as salient by LIME and a gradient-based method are being used by the classifier. |
| Approach: | They use a BERT BASE model to mask the sentiments of an English dataset and find that both methods produce faithful rationales. |
| Outcome: | The proposed classifier outperforms both the gradient-based and black-box saliency methods on the SemEval 2016 english dataset. |
ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks assess LLM performance in single-course settings and lack systematic evaluation in multi-course scenarios, where a patient’s condition evolves over time. |
| Approach: | They propose to use large language models to assess their performance in multi-course clinical decision-making scenarios where a patient’s condition evolves over time. |
| Outcome: | The proposed model includes 1,275 Chinese and 5,804 English samples across four stages from admission to discharge. |
Are they lovers or friends? Evaluating LLMs’ Social Reasoning in English and Korean Dialogues (2026.acl-long)
Copied to clipboard
Eunsu Kim, Junyeong Park, Juhyun Oh, Kiwoong Park, Seyoung Song, A. Seza Doğruöz, Alice Oh, Najoung Kim
| Challenge: | Existing studies on LLMs' ability to infer social relationships have limited results for Korean and English. |
| Approach: | They propose a social reasoning task based on a 1.1k-dialogue dataset in English and Korean sourced from movie scripts to evaluate LLMs' ability to infer the social relationships between speakers. |
| Outcome: | The proposed task evaluates the ability of LLMs to infer the social relationships between speakers in 1.1k-dialogue datasets in English and Korean. |
Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. |
| Approach: | They investigate the effectiveness of using Large Language Models to generate culturally relevant commonsense QA datasets for Indonesian and Sundanese languages using both LLMs and human annotators. |
| Outcome: | The proposed model generates 4.5K questions per language, compared with 4.5k for Indonesian and 4.5km for Sundanese. |